Object detection-based notification

ABSTRACT

Implementations of the subject technology provide object detection and/or classification for electronic devices. Object detection and/or classification can be performed using a radar sensor of an electronic device. The electronic device may be a portable electronic device. In some examples, object classification using a radar sensor can be based on an identification of user motion using radar signals and/or based on extraction of surface features from the radar signals. In some examples, object classification using a radar sensor can be based on time-varying surface features extracted from the radar signals. Surface features that can be extracted from the radar signals include a radar cross-section (RCS), a micro-doppler signal, a range, and/or one or more angles associated with one or more surfaces of the object.

TECHNICAL FIELD

The present description relates generally to object detection andclassification by electronic devices, including generating anotification based on the object detection and/or classification.

BACKGROUND

Detection, classification, and tracking of objects in a physicalenvironment is often performed using Light Detection and Ranging (LIDAR)sensors or computer vision techniques applied to capturedoptical-wavelength images. However, it can be difficult to detect orclassify some objects, such as spatially uniform or opticallytransparent objects, using these sensors.

BRIEF DESCRIPTION OF THE DRAWINGS

Certain features of the subject technology are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of thesubject technology are set forth in the following figures.

FIGS. 1 and 2 illustrate aspects of an example electronic device inaccordance with one or more implementations.

FIG. 3 illustrates an example of a physical environment of an electronicdevice in accordance with implementations of the subject technology.

FIG. 4 illustrates example device motions and radar signals of anelectronic device in a physical environment in accordance with one ormore implementations of the subject technology.

FIG. 5 illustrates a schematic diagram of an illustrative data flow forobject classification using radar signals in accordance withimplementations of the subject technology.

FIG. 6 illustrates a schematic diagram of an illustrative data flow forfeature extraction from radar signals in accordance with implementationsof the subject technology.

FIGS. 7-9 illustrate example surface features that can be extracted fromradar signals in accordance with implementations of the subjecttechnology.

FIG. 10 illustrates an example frequency-space feature that can beextracted from radar signals in accordance with implementations of thesubject technology.

FIG. 11 illustrates additional example surface features andfrequency-space features that can be extracted from radar signals inaccordance with implementations of the subject technology.

FIG. 12 illustrates a flow chart of example operations that may beperformed for object classification using user motion informationextracted from radar signals in accordance with implementations of thesubject technology.

FIG. 13 illustrates a flow chart of example operations that may beperformed for object classification using surface features extractedfrom radar signals in accordance with implementations of the subjecttechnology.

FIG. 14 illustrates a flow chart of example operations that may beperformed for object classification by a portable electronic devicehaving a radar sensor in accordance with implementations of the subjecttechnology.

FIG. 15 illustrates an example improvement to a dead reckoning operationbased on object detection, tracking, and/or classification using radardata in accordance with implementations of the subject technology.

DETAILED DESCRIPTION

The detailed description set forth below is intended as a description ofvarious configurations of the subject technology and is not intended torepresent the only configurations in which the subject technology can bepracticed. The appended drawings are incorporated herein and constitutea part of the detailed description. The detailed description includesspecific details for the purpose of providing a thorough understandingof the subject technology. However, the subject technology is notlimited to the specific details set forth herein and can be practicedusing one or more other implementations. In one or more implementations,structures and components are shown in block diagram form in order toavoid obscuring the concepts of the subject technology.

Implementations of the subject technology described herein provideradar-based object detection, tracking, and/or classification forelectronic devices. Based on the detection and/or classification of anobject, an electronic device may generate a notification or alert, suchas to alert a user of the device that the object is approaching thedevice and/or a user of the device (e.g., due to motion of the objectand/or motion of the user of the device). In one or moreimplementations, the radar-based object detection, tracking, and/orclassification can be based on detection of a motion characteristic ofthe device itself. For example, a motion characteristic of the devicemay be the result, in some use cases, of motion of a platform on whichthe device is moving and/or user motion of a user of the electronicdevice that is carrying, wearing, and/or otherwise moving with theelectronic device. In one or more implementations, the radar-basedobject detection, tracking, and/or classification can be based on anextraction of surface features of an object from radar signals. Asexamples, surface features can include a radar cross-section (RCS), amicro-doppler feature, a range, an azimuth, and/or an elevation of theobject. In one or more implementations, the radar-based objectdetection, tracking, and/or classification may be performed using aradar sensor in a portable electronic device.

An illustrative electronic device including a radar sensor is shown inFIG. 1 . In the example of FIG. 1 , electronic device 100 (e.g., aportable electronic device) has been implemented using a housing that issufficiently small to be portable and carried by a user (e.g.,electronic device 100 of FIG. 1 may be a handheld electronic device suchas a tablet computer or a cellular telephone or smartphone). As shown inFIG. 1 , electronic device 100 includes a display such as display 110,which may be mounted on the front of housing 106. Electronic device 100includes one or more input/output devices such as a touch screenincorporated into display 110, a button or switch such as button 104and/or other input output components disposed on or behind display 110or on or behind other portions of housing 106. Display 110 and/orhousing 106 include one or more openings to accommodate button 104, aspeaker, a sensor, a light source, and/or a camera.

In the example of FIG. 1 , housing 106 includes two openings 108 on abottom sidewall of housing. One or more of openings 108 forms a port foran audio component. For example, one of openings 108 may form a speakerport for a speaker disposed within housing 106 and another one ofopenings 108 may form a microphone port for a microphone disposed withinhousing 106. Housing 106, which may sometimes be referred to as a case,may be formed of plastic, glass, ceramics, fiber composites, metal(e.g., stainless steel, aluminum, etc.), other suitable materials, or acombination of any two or more of these materials.

The configuration of electronic device 100 of FIG. 1 is merelyillustrative. In other implementations, electronic device 100 may be acomputer such as a computer that is integrated into a display such as acomputer monitor, a laptop computer, a wearable device such as a smartwatch, a pendant device, or other wearable or miniature device, a mediaplayer, a gaming device, a navigation device, a computer monitor, atelevision, a headphone, an earbud, or other electronic equipment. Insome implementations, electronic device 100 may be provided in the formof a wearable device such as a smart watch or smart glasses or otherheadset. In one or more implementations, housing 106 may include one ormore interfaces for mechanically coupling housing 106 to a strap orother structure for securing housing 106 to a wearer.

As shown in the example of FIG. 1 , the electronic device 100 mayinclude various sensors, such as an image sensor of a camera 112, aradar sensor 116, and/or one or more other sensors such as sensors 115.For example, sensors 115 may be or may include an infrared sensor (e.g.,an infrared imaging sensor or a light detection and ranging (LIDAR)sensor), an inertial sensor (e.g., an inertial measurement unit (IMU)sensor, such as an accelerometer, a gyroscope, and/or a magnetometer),an ambient light sensor, or any other sensor for sensing aspects of thephysical environment. In the example of FIG. 1 , the camera 112 and asensor 115 are positioned on a front surface of the electronic device100 (e.g., a surface on which the display 110 is disposed), and theradar sensor 116 is depicted as being positioned on a rear surface ofthe electronic device. In this configuration the radar sensor 116 canemit radar signals and receive radar reflections of the radar signals infront of the user while the user is viewing the display. However, otherconfigurations are contemplated in which one or more cameras and/or oneor more other sensors are positioned on the rear surface, an edge,and/or another location on the electronic device, and/or one or moreradar sensors are positioned on the front surface, an edge, or anotherlocation on the electronic device.

In some examples, as illustrated in FIG. 2 , electronic device 100includes various components, such as processor(s) 190, RF circuitry(ies)103 (e.g., WiFi, Bluetooth, near field communications (NFC) or other RFcommunications circuitry), memory(ies) 107, image sensor(s) 111 (e.g.,image sensors of a camera such as camera 112, or other imaging sensors),inertial sensor(s) 113 (e.g., one or more accelerometers, one or moregyroscopes, and/or one or more magnetometers), microphone(s) 119, radarsensor(s) 189 (e.g., implementations of radar sensor 116), rangingsensor(s) 121 such as LIDAR sensors, speaker(s) 118, display 110, andtouch-sensitive surface(s) 122. These components optionally communicateover communication bus(es) 150 of electronic device 100.

In the example of FIG. 2 , electronic device 100 includes processor(s)190 and memory(ies) 107. Processor(s) 190 may include one or moregeneral processors, one or more graphics processors, and/or one or moredigital signal processors. In some examples, memory(ies) 107 may includeone or more non-transitory computer-readable storage mediums (e.g.,flash memory, random access memory, volatile memory, non-volatilememory, etc.) that store computer-readable instructions configured to beexecuted by processor(s) 190 to perform the techniques described below.

Electronic device 100 includes RF circuitry(ies) 103. RF circuitry(ies)103 optionally include circuitry for communicating with electronicdevices, networks, such as the Internet, intranets, and/or a wirelessnetwork, such as cellular networks and wireless local area networks(LANs). RF circuitry(ies) 103 optionally includes circuitry forcommunicating using near-field communication and/or short-rangecommunication, such as Bluetooth®.

Electronic device 100 may include one or more displays, such as display110. Display 110 may include an opaque display. Display 110 may includea transparent or semi-transparent display that may incorporate asubstrate through which light representative of images is directed to anindividual's eyes. Display 110 may incorporate LEDs, OLEDs, a digitallight projector, a laser scanning light source, liquid crystal onsilicon, or any combination of these technologies. The substrate throughwhich the light is transmitted may be a light waveguide, opticalcombiner, optical reflector, holographic substrate, or any combinationof these substrates. In one example, the transparent or semi-transparentdisplay may transition selectively between an opaque state and atransparent or semi-transparent state. Other examples of display 110include head up displays, automotive windshields with the ability todisplay graphics, windows with the ability to display graphics, lenseswith the ability to display graphics, tablets, smartphones, and desktopor laptop computers. Alternatively, electronic device 100 may bedesigned to receive an external display (e.g., a smartphone). In someexamples, electronic device 100 is a projection-based system that usesretinal projection to project images onto an individual's retina orprojects virtual objects into a physical setting (e.g., onto a physicalsurface or as a holograph).

In some examples, electronic device 100 includes touch-sensitivesurface(s) 122 for receiving user inputs, such as tap inputs and swipeinputs. In some examples, display 110 and touch-sensitive surface(s) 122form touch-sensitive display(s).

Electronic device 100 may include image sensor(s) 111. Image sensors(s)111 optionally include one or more visible light image sensors, such ascharged coupled device (CCD) sensors, and/or complementarymetal-oxide-semiconductor (CMOS) sensors operable to obtain images ofphysical elements from the physical setting. Image sensor(s) alsooptionally include one or more infrared (IR) sensor(s), such as apassive IR sensor or an active IR sensor, for detecting infrared lightfrom the physical setting. For example, an active IR sensor includes anIR emitter, such as an IR dot emitter, for emitting infrared light intothe physical setting. Image sensor(s) 111 also optionally include one ormore event camera(s) configured to capture movement of physical elementsin the physical setting. Image sensor(s) 111 also optionally include oneor more depth sensor(s) configured to detect the distance of physicalelements from electronic device 100. In some examples, electronic device100 uses CCD sensors, event cameras, and depth sensors in combination todetect the physical setting around electronic device 100.

In some examples, radar sensor(s) 189 may include one or more millimeter(MM) wave radar sensors and/or one or more radar sensors configured toemit radar signals and receive reflected radar returns in a frequencyrange between 40 gigahertz (GHz) and 100 GHz (e.g., between 55 GHz and65 GHz between 75 GHz and 82 GHz), between 26.5 GHz and 40 GHz, between18-26.5 GHz, between 12.5-18 GHz, between 8-12.5 GHz, between 4-8 GHz,between 2-4 GHz, between 1-2 GHz, or between 0.3-1 GHz and/or awavelength of between 0.75-0.30 cm, (e.g., between 5.45 mm and 4.61 mmor between 3.7 mm and 3.9 mm), between 11-7.5 mm, between 17-11 mm,between 24-17 mm, between 37.5-24 mm, between 75-37.5 mm, between 150-75mm, between 300-150 mm, or between 1000-300 mm (as examples). Forexample, in or more implementations, radar sensor(s) 189 (e.g.,including radar sensor 116) may include a mm wave transceiver configuredto emit radar signals (e.g., millimeter wavelength electromagneticwaves), and to receive and detect reflections of the emitted radarsignals from one or more objects in the environment around theelectronic device 100. In one or more implementations, a mm wave radarsensor may be implemented in radar sensor(s) 189 to provide improvedaccess to doppler characteristics in the radar returns (e.g., relativeto other radar sensors and/or non-radar sensors).

In some examples, electronic device 100 includes microphones(s) 119 todetect sound from the user and/or the physical setting of the user. Insome examples, microphone(s) 119 includes an array of microphones(including a plurality of microphones) that optionally operate intandem, such as to identify ambient noise or to locate the source ofsound in space of the physical setting.

Electronic device 100 may also include inertial sensor(s) 113 fordetecting orientation and/or movement of electronic device 100 and/orthe radar sensor(s) 189. For example, electronic device 100 may useinertial sensor(s) 113 to track changes in the position and/ororientation of electronic device 100, such as with respect to physicalelements in the physical environment around the electronic device 100.Inertial sensor(s) 113 may include one or more gyroscopes, one or moremagnetometers, and/or one or more accelerometers.

FIG. 3 illustrates an example physical environment in which anelectronic device such as electronic device 100 may be implementedand/or operated, according to aspects of the disclosure. In the exampleof FIG. 3 , a physical environment 300 of the electronic device 100includes a ground surface 304 (e.g., a floor or an outdoor groundsurface), a planar surface 302 (e.g., a wall, a door, and/or a window),and physical object 306. The physical object 306 may be a stationaryobject (e.g., a column, a beam, a piece of furniture, an appliance,etc.) or a moving object (e.g., a person or an animal walking, running,or playing, or a moving machine such as a vehicle). As illustrated inFIG. 3 , radar sensor 116 of the electronic device 100 may emit radarsignals 303 and receive reflections of portions of the emitted radarsignals 303 from various objects and/or surfaces in the physicalenvironment 300, including the ground surface 304, the planar surface302, and/or the physical object 306. Radar signals 303 may beelectromagnetic waves having radar frequencies and wavelengths, such aselectromagnetic waves having a frequency of between 40 gigahertz (GHz)and 100 GHz (e.g., between 55 GHz and 65 GHz between 75 GHz and 82 GHz),between 26.5 GHz and 40 GHz, between 18-26.5 GHz, 12.5-18 GHz, 8-12.5GHz, 4-8 GHz, 2-4 GHz, 1-2 GHz, or 0.3-1 GHz and/or a wavelength ofbetween 0.75-0.30 cm, (e.g., between 5.45 mm and 4.61 mm or between 3.7mm and 3.9 mm), 11-7.5 mm, 17-11 mm, 24-17 mm, 37.5-24 mm, 75-37.5 mm,150-75 mm, 300-150 mm, or 1000-300 mm (as examples).

In the example of FIG. 3 , a single ground surface 304, a single planarsurface 302, and a single physical object 306 that reflect the radarsignals 303 from the radar sensor 116. However, it is appreciated thatthe physical environment 300 may include any number of ground surfaces,any number of planar and/or non-planar surfaces, and/or any number ofstationary and/or moving physical objects 306 that can reflect the radarsignals from the radar sensor 116, and can be detected, tracked, and/orclassified using the reflected radar signals.

In one or more implementations, the electronic device 100 may also movewithin the physical environment 300. For example, a user of theelectronic device 100 may carry or wear the electronic device 100 whilemoving (e.g., walking, running, or otherwise moving) within the physicalenvironment 300.

For example, FIG. 4 illustrates an example use case in which a user 101is carrying the electronic device 100 while walking in a direction 403toward a door 400. As illustrated in FIG. 4 , while the user 101 iswalking in direction 403 while holding electronic device 100, theelectronic device 100 may transmit a radar signal 402, and receive areflected portion 404 of the transmitted radar signal 402, reflected bythe door 400. Although not explicitly depicted in FIG. 4 , otherportions of the transmitted radar signal 402 may be reflected from aground surface 406 on which the user 101 is walking, and/or othersurfaces and/or objects, such as a wall 411 around the door 400, and/ora window 413.

As illustrated in FIG. 4 , the electronic device 100 itself may movewith various motions due to motions and/or vibrations of a platform onwhich the electronic device is moving, such as due to the walking motionof the user 101 in this example. As examples, the electronic device 100may experience a vertical oscillatory motion 405 due to the motion ofthe user's body as the user walks. In implementations in which theelectronic device 100 is held in the user's hand or worn on the user'swrist, the electronic device 100 may also experience a verticaloscillatory motion 407 and a horizontal oscillatory motion 409 due to anarm swinging motion of the user as the user walks. In one or moreimplementations, the electronic device 100 may experience and/or detectadditional motions such as motions due to the leg swinging motion of theuser as the user walks. In one or more implementations, time variationsin the reflected portions 404 of the transmitted radar signal 402 due tothe oscillatory motions 405, 407, and/or 409 can be used to classifyand/or distinguish objects such as the door 400 and the wall 411. Forexample, in one or more implementations, time variations in surfacefeatures extracted from the reflected portions 404 of the transmittedradar signal 402, such as a time varying RCS, a time varyingmicro-doppler signal, and/or a time varying range, azimuth, and/orelevation can be used for detection, tracking, and/or classification ofan object, such as the door 400 and/or the wall 411. In one or moreimplementations, tracking of device motion using an inertial sensor suchas inertial sensor(s) 113 (see FIG. 2 ) may be used in combination withthe radar signals to detect, track, and/or classify objects in thephysical environment 300. In one or more implementations, the electronicdevice 100 may determine that the electronic device 100 is on acollision course with a detected and/or classified object, and generatea notification or an alert of a potential upcoming collision. In one ormore implementations, the electronic device 100 may generate anotification or an alert other than a collision alert, such as an alertto indicate the presence of a particular object or type of object. Inone or more implementations, radar signals may be used to correct orimprove device motion tracking information generated by the inertialsensor(s).

FIG. 5 illustrates an example data processing flow that may be performedfor detection, tracking, and/or classification of objects using radarsignals from a radar sensor of an electronic device. In the example ofFIG. 5 , a wireless transceiver (e.g., an implementation of radar sensor116) provides radar signals to a target detection module 502. In one ormore implementations, the wireless transceiver 500 may be a millimeterwave radar transceiver, such as a 60 GHz millimeter wave radartransceiver, that is configured to emit radar signals and receiveportions of the radar signals that are reflected by various objects inthe physical environment of the transceiver. The radar signals mayinclude raw radar signals from the wireless transceiver 500 and/or mayinclude range data, azimuth data, and/or elevation data associated withone or reflections.

The target detection module 502 may process the received radar signalsand generate, for example, a point-cloud that contains points for alldetected objects in the field of view of the wireless transceiver 500.The point cloud may be provided (e.g., as potential targets), totarget-of-interest (TOI) identifier 504. As shown, an inertial sensor506 (e.g., an implementation of inertial sensor(s) 113) may also provideinertial data (e.g., position and/or motion data for the wirelesstransceiver 500 and/or the electronic device 100 based on accelerometerdata, gyroscope data, and/or magnetometer data) to thetarget-of-interest identifier 504. Using the point cloud from the targetdetection module 502 and inertial data from the inertial sensor 506, asubset of detected targets can be identified as targets of interest bythe TOI identifier 504. For example, using the inertial data, the TOIidentifier 504 may determine a direction of movement of the device, andidentify targets in the point cloud that are located within a projectedpath of the device (e.g., within a bore-sight field of view of a devicein one implementation, or within an angular range of the projected pathin various other implementations), from among the targets detected bythe target detection module 502, as targets of interest. As indicated inFIG. 5 , target locations for the targets of interest may be providedfrom the TOI identifier 504 to a feature extractor 508.

In the example of FIG. 5 , for a given target of interest that isidentified by the TOI identifier 504, the feature extractor 508 mayextract object feature data 510 (e.g., surface features, includingstatic features and/or or time-varying features) from the radar signals.Examples of object feature data 510 (e.g., surface features) that can beextracted from radar signals include a time-varying radar cross-section(RCS), a micro-doppler feature, and high-resolution range/angleinformation. For example, common objects that are found in a physicalenvironment often exhibit unique structural features that can induce atime-varying signature on the radar returns at a moving electronicdevice (e.g., an electronic device that is being carried or worn by auser moving around the physical environment). As examples, glass doorsare often formed from a single pane of glass, windows are often formedfrom double pained glass, and interior walls are often formed frommulti-layered sheetrock. As the wireless transceiver 500 approaches anobject (e.g., while implemented in a moving electronic device, such aselectronic device 100), the structure of the object (e.g. one or moresurfaces of one or more glass layers) may impose a multipath conditionon the reflected signal that varies with time due toconstructive/destructive interference by reflections from differentsurfaces. The signal reflected from the object can be processed toextract the (e.g., time-varying) signature that represents the physicalstructure.

As shown, the object feature data 510 extracted by the feature extractor508 may be provided to a classifier 512. In one or more implementations,classifier 512 may include a machine learning model 514 that has beentrained to classify objects based on object feature data obtained, atleast in part, on radar signals (e.g., radar reflections). For example,responsive to providing an input including a given set of measuredsurface features (e.g., object feature data 510), the classifier 512(e.g., machine learning model 514) can provide an output that includesobject information for a detected object of interest. In one or moreimplementations, the object information may include an object type.Example object types that may be identified by the classifier mayinclude a standing human, a sitting human, a walking human, a runninghuman, a glass wall, a glass window, a wooden wall, a wooden door, asheetrock wall, a sheetrock door. In one or more implementations, theclassifier 512 may also output a classification probability orconfidence level that indicates the confidence with which the object hasbeen classified.

In the example of FIG. 5 , a captured radar signal is processed todetect and classify one or more objects. At the output of FIG. 5 ,object information for one or more detected objects can be provided toone or more higher layer applications (e.g., a collision warningapplication, another notification or alert system, a dead reckoningsystem, etc.) at the device or at another device or system for furtherprocessing, as described herein. For each detected object of interest,the object information may include, for example, a range to the object,one or more angles (e.g., an azimuth and/or an elevation), a speed orvelocity, and/or an object type. Detecting an object may includedetecting a reflected radar signal and determining a location from whichthe radar signal was reflected (e.g., using target detection module 502and/or TOI identifier 504). Classifying a detected object may includedetermining (e.g., using classifier 512) the object type and/or anobject material of the detected object (e.g., determining that thedetected object is a wall, a window, a person, and/or formed from glass,concrete, wood, other materials in one or more layers such as planar ornon-planar layers). In one or more implementations, the probability orconfidence level that is output by the classifier with an object typemay be used by an application or process at the electronic device 100 todetermine whether to take action (e.g., provide an alert, or display aclassification) based on the identified object type.

FIG. 6 illustrates additional details of the target detection and TOIidentification operations of FIG. 5 , in accordance with one or moreimplementations. In the example of FIG. 6 , radar data (e.g., a radardata cube from the wireless transceiver 500 of FIG. 5 , the radar datacube including processed received signals that contain range, angleand/or doppler information for all potential targets) may be provided toa target location estimator 600 (e.g., implemented as or as part of thetarget detection module 502 of FIG. 5 ). In one or more implementations,the target location estimator 600 may estimate target locations for allpotential targets by performing a spectral estimation operation (e.g., a2D fast Fourier transform (FFT)). The output of the target locationestimator 600 may be provided to a Constant False Alarm Rate (CFAR)detector to extract the detection point cloud described above inconnection with FIG. 5 . As shown in FIG. 6 , the target detections(e.g., the point cloud) may be passed from the target location estimator600 to a data association and tracking module 602. The data associationand tracking module 602 may track one or more of the detected targetsover time, as objects. As shown, the data association and trackingmodule 602 may provide a target location estimate for each trackedobject at a time-sample, to a beam former 606. The beam former 606 mayapply beam forming to the radar data cube using the provided targetlocation, to isolate a given target corresponding to the target locationfrom the other targets detected by the target location estimator 600.Using the beam formed radar signals, for each tracked target, the RCS, amicro-doppler signal, and a high-resolution range-azimuth may beestimated for each frame or sample time. As shown, the object featuredata 510 of a given target may then be passed to the classifier 512(e.g., to the machine learning model 514) to classify the detectedobject.

In the examples of FIGS. 5 and 6 , the elements indicated with dashedboxes (e.g., the target detection module 502, the TOI identifier 504,the feature extractor 508, the classifier 512, the machine learningmodel 514, the target location estimator 600, the data association andtracking module 602, the object tracker 604, and the beam former 606)may be implemented in hardware, software (e.g., stored in memory(ies)107 and executed by processor(s) 190), or a combination of hardware andsoftware. In the examples of FIGS. 5 and 6 , the elements indicated bydashed boxes (e.g., the target detection module 502, the TOI identifier504, the feature extractor 508, the classifier 512, the machine learningmodel 514, the target location estimator 600, the data association andtracking module 602, the object tracker 604, and the beam former 606)are indicated as being implemented separately. In other implementations,one or more or all of the target detection module 502, the TOIidentifier 504, the feature extractor 508, the classifier 512, themachine learning model 514, the target location estimator 600, the dataassociation and tracking module 602, the object tracker 604, and thebeam former 606 may be implemented as parts of a common processingmodule. In the examples of FIGS. 5 and 6 , the elements indicated onlywith solid rectilinear boxes may include at least one or more hardwarecomponents, such as sensing elements and/or emitters, which may beimplemented using antennas and/or readout circuitry.

FIGS. 7, 8, and 9 illustrate examples of surface features (e.g.,examples of the extracted surface features that may be included in theobject feature data 510) that can be extracted from radar signals andused for object detection, classification, and/or tracking. As oneexample, FIG. 7 illustrates a graph 700 of RCS measurements versus rangeto the reflecting object taken over time. The RCS measurements of FIG. 7can be made using a radar sensor, such as the radar sensor 116 ofelectronic device 100, in an example use case. In the example of FIG. 7, radar reflections received by the radar sensor include firstreflections from a ground surface (e.g., the floor under the device) andsecond reflections from a glass door, received while the electronicdevice including the radar sensor is being carrier or worn by a userthat is walking toward the glass door. As shown, when the RCS and therange of detected objects corresponding to the floor and the glass doorare generated over time as the range to the floor remains substantiallyconstant and the range to the glass door changes, the RCS measurements702 of the floor and the RCS measurements 704 of the glass door aredistinct from each other, and have different variation patterns. Thus,the RCS variations over time can be used (e.g., using machine learningmodel 514 trained using RCS variation training data) to identify theobject being tracked as a glass door.

FIG. 8 illustrates a graph 800 of a micro-doppler feature measured usinga radar sensor, such as the radar sensor 116 of an electronic device100, in the same example use case described above in connection withFIG. 7 . In the example of FIG. 8 , the velocity of the glass wall,relative to the electronic device 100, is measured using the dopplershift of the radar reflections from the glass wall. As shown in FIG. 8 ,the measured velocity of the glass door includes a mean value 802 thatis doppler shifted from mean zero 803 by an amount that corresponds tothe mean speed of the walking user. As shown, the measured velocity ofthe glass door also includes micro-doppler features such as oscillations804 that oscillate with a frequency that corresponds to the cadence ofthe device user's gait.

FIG. 9 illustrates a power-range profile 900 that shows the power 902(e.g., which, during device use, can be measured using a radar sensor,such as the radar sensor 116 of an electronic device 100), versus therange to an object (e.g., a glass door in a use case similar to theexample use case described above in connection with FIGS. 7 and 8 ). Inthe example of FIG. 9 , the power (e.g., of the reflected radar signal)generally increases with reduced range to the glass door, and representsthe power-range profile of both the surface of the glass door andinternal structure reflections from multiple surfaces of the glass door.For example, the power-range profile of a glass door formed from asingle pane of glass (e.g., with two outer surfaces that interface withthe air) may be different from the power-range profile of glass windowformed from multiple panes of glass, a sheetrock wall, a piece offurniture, a column, or a person in a way that is distinguishable by theclassifier 512 (e.g., the machine learning model 514). For example, theranges extracted from the radar signals for the power-range profile maybe extracted with sufficiently high resolution for differences in thesurface smoothness and/or the number of surfaces that form the object toaffect the shape of the power-range profile. In one or moreimplementations, the shape of the power-range profile may be used (e.g.,alone or in combination with the micro-doppler feature and/or the RCS)to classify or otherwise identify a detected object.

For example, when a target of interest (TOI) is identified by anelectronic device, one or more high-resolution feature extractionprocesses at the electronic device can provide a high-resolutionestimate of the power-range profile of the TOI. For example,beam-forming operations can be applied to radar returns from glass toisolate two or more separate surface reflections. Then, applying ahigh-resolution feature extraction process to the beam formed returns,the electronic device can estimate the power-range profile of themultiple internal reflections of the object (e.g. internal reflectionsof two surfaces of single-pained glass or four surfaces of double-painedglass). In this way, a high-resolution power-range profile of the TOI,such as is illustrated in FIG. 9 , can then be used as a feature in theclassification process of the TOI.

Although a power-range profile is illustrated in FIG. 9 , it isappreciated that power-angle profiles and/or range-angle profilesextracted from radar returns can also be used in the classification ofobject. For example, power-angle profiles may include a power-azimuthprofile and/or a power-elevation profile. For example, a range-angleprofile may include a range-azimuth profile and/or a range-elevationprofile. In one or more implementation, a power-range-azimuth profileand/or a power-range-elevation profile may also be extracted from radarreturns and used to classify one or more objects. For example, the timevariation of any or all of a power-azimuth profile, a power-elevationprofile, a range-azimuth profile, a range-elevation profile, apower-range-azimuth profile, and/or a power-range-elevation profile mayvary in a way that is characteristic of a particular object and/orobject type.

In the examples of FIGS. 7-9 , features that may be used to classify anobject using radar data are illustrated in time-space. In one or moreimplementations, frequency-space data generated from radar signals canalso, or alternatively be used for object detection, tracking and/orclassification. As an example, FIG. 10 illustrates a transform 1000 ofthe micro-doppler feature of FIG. 8 , in which the cadence 1002 can beseen, and can be seen to be substantially constant over time. Thecadence 1002 may correspond to the frequency of the oscillations 804 ofFIG. 8 , and to the cadence of the user's steps as the user walks towardthe glass door.

In one or more implementations, object detection, tracking, and/orclassification can also include comparisons of radar features ofmultiple objects. For example, FIG. 11 includes a micro-doppler graph1100, and a frequency transform 1108, extracted from radar reflectionsfrom the floor, while the user of the example of FIGS. 7-10 is walkingtoward the glass door. As shown, the mean velocity 1102 of the floor,measured using the doppler shift of the radar reflections from the floorhas a value of substantially zero (e.g., due to the substantiallyconstant mean height of the device being carried or worn by the walkinguser). As shown, the micro-doppler feature of the floor also includesoscillations 1104 that oscillate with a frequency that corresponds tothe cadence of the user's gait. In the example of FIG. 11 , additionalvelocity oscillations 1106 can also be seen that correspond to theleg-swing motion of the user's legs while walking. In one or more usecases, arm-swing motions of the user's arms while walking may alsogenerate additional oscillations in the micro-doppler features. Thefrequency transform 1108 of FIG. 11 also shows a first cadence 1112 thatcorresponds to the frequency of the oscillations 1104, and a secondcadence 1110 that corresponds to the oscillations 1106.

In one or more implementations, by comparing the micro-doppler featureof a ground reflection with the micro-doppler feature of the incomingglass wall of the example of FIGS. 7-10 , it is possible to see thatthere is a high correlation between the micro-doppler features. In oneor more implementations, the electronic device 100 may determine thatthe glass wall (or another stationary object) is stationary, bydetermining that the cadence 1002 is substantially the same as (e.g.,within a predetermined similarity threshold) the cadence 1112. As analternative example, radar reflections of a moving object, such asanother person walking toward or away from the user, would includeoscillations as the cadence of the gait of the other person and/or armswing motions of the other person, which would have a lower or nocorrelation with the cadence(s) of the user of the electronic device100.

FIG. 12 illustrates a flow diagram of an example process 1200 forproviding object detection, tracking, and/or classification inaccordance with implementations of the subject technology. Forexplanatory purposes, the process 1200 is primarily described hereinwith reference to the electronic device 100 of FIGS. 1 and 2 . However,the process 1200 is not limited to the electronic device 100 of FIGS. 1and 2 , and one or more blocks (or operations) of the process 1200 maybe performed by one or more other components of other suitable devicesor systems. Further for explanatory purposes, some of the blocks of theprocess 1200 are described herein as occurring in serial, or linearly.However, multiple blocks of the process 1200 may occur in parallel. Inaddition, the blocks of the process 1200 need not be performed in theorder shown and/or one or more blocks of the process 1200 need not beperformed and/or can be replaced by other operations.

As illustrated in FIG. 12 , at block 1202, an electronic device (e.g.,electronic device 100) may obtain radar signals from a radar sensor(e.g., radar sensor 116) of the electronic device. Obtaining the radarsignals may include emitting, with the radar sensor, radar signals, andreceiving reflected portions of the emitted signals. The reflectedportions may be reflected from one or more objects in the physicalenvironment of the electronic device.

At block 1204, the electronic device may identify a motioncharacteristic corresponding to the electronic device based on the radarsignals. The motion characteristic may include, as examples,characteristics of a walking motion, a leg swing motion, and/or an armswing motion. In one or more implementations, identifying the motioncharacteristic may include identifying, using the radar signals (e.g.,using feature extractor 508), a first cadence corresponding to thewalking motion.

At block 1206, the electronic device may detect an object in anenvironment of the electronic device using the radar signals. Forexample, detecting the object may include performing any or all of theoperations described herein in connection with the target detectionmodule 502, the TOI identifier 504, the feature extractor 508, thetarget location estimator 600, the data association and tracking module602, and the object tracker 604 of FIGS. 5 and 6 (e.g., to determine thepresence of an object based on reflected radar signals and to determinea location of the reflecting object).

At block 1208, the electronic device may classify (e.g., with theclassifier 512) the object using the radar signals and the identifiedmotion characteristic. In one or more implementations, classifying theobject may include classifying the object as a moving object or astationary object. In one or more implementations, classifying theobject may also, or alternatively, include determining an object typefor the object. As examples, an object type may be a human, a glasswall, a glass window, a wooden door, a sheetrock wall, etc. In one ormore implementations, identifying the object type may include extractingsurface features from the radar signals, and providing the surfacefeatures to a classifier, such as the classifier 512 of FIGS. 5 and 6 ,and obtaining an output of the classifier. For example, providing thesurface features to the classifier may include providing the surfacefeatures to a machine learning model, such as machine learning model514, that has been trained to identify object types based on surfacefeatures extracted from radar signals.

For example, in a use case in which the object is stationary object,classifying the object may include identifying, using the radar signals(e.g., using feature extractor 508), a second cadence corresponding tothe object; and determining (e.g., with the classifier 512) that thesecond cadence substantially matches the first cadence. For example, asdiscussed above in connection with FIGS. 4, 8, 10, and 11 , the secondcadence may correspond to an oscillatory motion of the electronic devicewith respect to the stationary object, due to the motion of the userthat is carrying or wearing the electronic device. For example, in oneor more implementations, identifying the first cadence using the radarsignals may include identifying the first cadence using a first portionof the radar signals corresponding to a reflection from a groundsurface, and identifying the second cadence using the radar signals mayinclude identifying the second cadence using a second portion of theradar signals corresponding to a reflection from the object.

As discussed herein, in one or more implementations, detection andtracking of objects that have been determined to be stationary objectsusing a radar sensor can be used to improve and/or correct tracking ofdevice motion, as determined by other sensors, such as GPS sensorsand/or IMU sensors. In one or more implementations, the electronicdevice may track motion of the electronic device using a sensor of theelectronic device other than the radar sensor. The electronic device maydetermine a location of the stationary object using the radar signals,and modify (e.g., correct) the tracking of the motion of the electronicdevice based on the location of the stationary object.

In one use case, the object may be a stationary planar object. In one ormore use cases, the stationary planar object may include a pane of glass(e.g., a pane of glass that forms or is part of a glass wall, a glassdoor, or a window). In one or more implementations, classifying theobject at block 1208 may include classifying the object as glass (e.g.,using the machine learning model 514 or another classification enginethat is configured to distinguish between glass, wood, sheetrock, and/ormetal planar surfaces using radar signals, as described herein).

In one or more implementations, identifying the motion characteristicmay include determining a velocity of the electronic device (e.g., andthe user carrying or wearing the electronic device) relative to the paneof glass.

At block 1210, the electronic device may determine whether to generatean alert based on the detecting and classifying of the object. In one ormore implementations, the electronic device may determine whether togenerate the alert, at least in part, by determining a time-to-impactbetween the electronic device and the pane of glass based on thevelocity.

For example, the electronic device may determine whether to generate thealert, e.g., a collision alert, at least in part by determining that thetime-to-impact satisfies (e.g., is less than and/or equal to) athreshold for generating the alert. The electronic device may generatethe alert, for example, responsive to determining that thetime-to-impact satisfies the threshold. In this way, the electronicdevice can help the device and/or a device user of the device to avoidcollisions with glass doors or glass walls, or other glass or otheroptically transparent objects that may be difficult for the deviceand/or the user to visually detect alone, in one or moreimplementations.

Although the example discussed above describes a use case in which theobject is a stationary object, in other use cases, the object may be amoving object (e.g., another person walking near the electronic deviceor another moving object). In a use case in which the object is a movingobject, classifying the object may include identifying, using the radarsignal, a second cadence corresponding to the object, and determiningthat the second cadence is different from the first cadence. Forexample, the second cadence may correspond to a walking motion ofanother person and/or an arm swing motion or other motion of the otherperson, that differs from the cadence of the motion(s) of the user ofthe electronic device in frequency, phase, and/or amplitude. In this usecase, identifying the first cadence using the radar signals may includeidentifying the first cadence using a first portion of the radar signalscorresponding to a reflection from a ground surface, and identifying thesecond cadence using the radar signals may include identifying thesecond cadence using a second portion of the radar signals correspondingto a reflection from the object, the object being different from theground surface.

In one or more implementations, motion characteristic corresponding tothe electronic device may be the result of one or more characteristicsof user motion of a device user of the electronic device. For example,the electronic device may determine, based on the identified motioncharacteristic, a stride length of the user (e.g., based in part on thefirst cadence). In one or more implementations, the electronic devicemay also generate health data for the user based on the stride length.For example, in one or more implementations, the electronic device maydetermine a step count corresponding to a number of steps taken by theuser based on the radar signals. The electronic device may alsodetermine a distance traveled by the user based on the radar signalsand/or other sensor signals. For example, the distance traveled may bedetermined using an inertial sensor and/or a GPS sensor of theelectronic device. In one or more implementations, the electronic devicemay modify (e.g., improve or correct) the traveled distance using thedetermined stride length and/or other radar signal data. In one or moreother implementations, the electronic device may determine the distancetraveled directly from the radar data and independently of other sensordata (e.g., independently of inertial sensor data).

FIG. 13 illustrates a flow diagram of another example process 1300 forproviding object detection, tracking, and/or classification inaccordance with implementations of the subject technology. Forexplanatory purposes, the process 1300 is primarily described hereinwith reference to the electronic device 100 of FIGS. 1 and 2 . However,the process 1300 is not limited to the electronic device 100 of FIGS. 1and 2 , and one or more blocks (or operations) of the process 1300 maybe performed by one or more other components of other suitable devicesor systems. Further for explanatory purposes, some of the blocks of theprocess 1300 are described herein as occurring in serial, or linearly.However, multiple blocks of the process 1300 may occur in parallel. Inaddition, the blocks of the process 1300 need not be performed in theorder shown and/or one or more blocks of the process 1300 need not beperformed and/or can be replaced by other operations.

As illustrated in FIG. 13 , at block 1302, an electronic device (e.g.,electronic device 100) may obtain radar signals from a radar sensor(e.g., radar sensor 116) of the electronic device. Obtaining the radarsignals may include emitting, with the radar sensor, radar signals, andreceiving reflected portions of the emitted signals. The reflectedportions may be reflected from one or more objects in the physicalenvironment of the electronic device.

At block 1304, the electronic device (e.g., feature extractor 508) mayextract a radar cross-section (RCS) and a micro-doppler signal from theradar signals. In one or more implementations, the electronic device mayalso extract a range and an angle (e.g., an azimuth and/or an elevation)for the object from the radar signals.

At block 1306, the electronic device may classify (e.g., with classifier512 and/or machine learning model 514) an object in an environment ofthe electronic device based on the radar cross-section and themicro-doppler signal. In one or more implementations, classifying theobject based on the radar cross-section and the micro-doppler signal mayinclude classifying the object based on a time variation of the radarcross-section (e.g., as described in connection with the example of FIG.7 ) and a time variation of the micro-doppler signal (e.g., as describedin connection with the example of FIGS. 8, 10 and 11 ). In one or moreimplementations, the signal reflected from the object can be processedto extract the signature that represents the physical structure. Forexample, in one or more implementations, classifying the object based onthe time variation of the radar cross-section and the micro-dopplersignal may include distinguishing between an opaque planar surface and atransparent planar surface using at least the time variation of theradar cross-section. For example, as the radar sensor approaches theobject, the structure of the object (e.g. internal glass layers) mayimpose a multipath condition that varies with time due toconstructive/destructive interference.

As described herein, in one or more implementations, classifying theobject may include providing the radar cross-section and themicro-doppler signal to a machine learning engine (e.g., machinelearning model 514) at the electronic device, and obtaining an objectclassification as an output from the machine learning engine.

In one or more implementations, prior to classifying the object at block1306, the electronic device (e.g., target detection module 502 and/ortarget location estimator 600) may perform an initial target detectionusing the radar signals. The electronic device (e.g., TOI identifier 504and/or object tracker 604) may determine a location of a potentialtarget object based on the initial target detection. The electronicdevice (e.g., feature extractor 508 and/or beam former 606) may extractthe radar cross-section and the micro-doppler signal (e.g., and a rangeand/or one or more angles) from the radar signals based on the location(e.g., as described above in connection with FIGS. 5 and 6 ). In one ormore implementations, extracting the radar cross-section and themicro-doppler signal (e.g., and a range and/or one or more angles) fromthe radar signals based on the location may include performing a beamforming operation on the radar signals using the location (e.g., asdescribed above in connection with FIG. 6 ).

At block 1308, the electronic device may determine, based at least inpart on the classification of the object, whether to generate an alert.For example, the electronic device may determine that the object is astationary object or a moving object (e.g., and/or whether the object isan visible opaque object or a transparent object such as a window or aglass wall or door) and determine whether to generate the alert bydetermining a velocity of the device relative to the object, and atime-to-impact between the device and the object based on the velocity.In one or more implementations, if the velocity and/or thetime-to-impact satisfy a threshold, the electronic device may determinethat the alert is to be generated, and generate the alert. As discussedherein, an alert generated by the electronic device may include anauditory alert, a tactile alert, and/or a visual alert.

FIG. 14 illustrates a flow diagram of an example process 1400 forproviding object detection, tracking, and/or classification inaccordance with implementations of the subject technology. Forexplanatory purposes, the process 1400 is primarily described hereinwith reference to the electronic device 100 of FIGS. 1 and 2 . However,the process 1400 is not limited to the electronic device 100 of FIGS. 1and 2 , and one or more blocks (or operations) of the process 1400 maybe performed by one or more other components of other suitable devicesor systems. Further for explanatory purposes, some of the blocks of theprocess 1400 are described herein as occurring in serial, or linearly.However, multiple blocks of the process 1400 may occur in parallel. Inaddition, the blocks of the process 1400 need not be performed in theorder shown and/or one or more blocks of the process 1400 need not beperformed and/or can be replaced by other operations.

As illustrated in FIG. 14 , at block 1402, a portable electronic devicethat includes a radar sensor may obtain a radar signal from a radarsensor (e.g., radar sensor 116). For example, the portable electronicdevice may be a handheld electronic device (e.g., a smartphone or atablet) or a wearable electronic device (e.g., a smart watch or smartglasses). Obtaining the radar signals may include emitting, with theradar sensor, radar signals, and receiving reflected portions of theemitted signals. The reflected portions may be reflected from one ormore objects in the physical environment of the electronic device.

At block 1404, the portable electronic device (e.g., TOI identifier 504and/or object tracker 604) may identify a target of interest in anenvironment of the portable electronic device using the radar signal.For example, in one or more implementations, the portable electronicdevice may include another sensor (e.g., an inertial sensor, such asinertial sensor(s) 113), and the portable electronic device may identifythe target of interest using the radar signal and sensor data from theother sensor of the portable electronic device.

At block 1406, the portable electronic device (e.g., feature extractor508 and/or beam former 606) may extract a surface feature (e.g., one ormore extracted surface features in the object feature data 510), for thetarget of interest, from the radar signals. The surface feature may be atime-varying surface feature. For example, the surface feature mayinclude at least one of a radar cross-section (RCS), a micro-dopplerfeature, or a range. The surface feature may also, or alternatively,include an angle such as an azimuth angle and/or an elevation angle.

At block 1408, the portable electronic device (e.g., classifier 512and/or machine learning model 514) may obtain a classification of anobject corresponding to the target of interest using the extractedsurface feature. In one or more implementations, the portable electronicdevice may obtain the classification using time-varying surfacefeatures. As examples, the electronic device may extract and use atime-varying micro-doppler feature, a time-varying range, a time-varyingRCS, a time-varying power-range profile, a range-angle profile, and/or atime-varying power-angle profile (e.g., a time-varying power-azimuthprofile, a time-varying power-elevation profile, a time-varyingpower-range-azimuth profile and/or a time-varyingpower-range-azimuth-elevation profile) for the classification. Forexample, in one exemplary use case, a human at long range approachingthe electronic device may have a motion characteristic of a single pointtarget in a range-azimuth point cloud. In this exemplary use case, asthe human approaches the electronic device, the range-azimuth pointcloud may spread to multiple spatial detections (e.g., highlightinghuman characteristic features of the human that differ from thecharacteristics of a point target).

In another exemplary use case, an approaching wall or pole (e.g., oranother stationary object that does not have separately moving parts)may exhibit less azimuth spread in a range-azimuth point cloud, than anapproaching human. Thus, a range-azimuth profile, such as a time-varyingrange-azimuth profile, can, in one or more use cases, further augmentother time-varying features, such as a time-varying RCS and/or atime-varying micro-doppler feature or a cadence, to enable moreeffective classification of objects.

In one or more implementations, the portable electronic device may alsoinclude a memory (e.g., memory(ies) 107) storing a machine learningmodel (e.g., machine learning model 514) trained to classify objectsbased on time-varying radar cross-sections (see, e.g., FIG. 7 ). Theextracted surface feature may include a radar cross-section, and theportable electronic device may obtain the classification of the objectcorresponding to the target of interest using the extracted surfacefeature by providing a time variation of the radar cross-section (e.g.,multiple RCS measurements obtained over time) to the machine learningmodel, and obtaining a resulting output from the machine learning model.In one or more implementations, additional time-varying information(e.g., a time-varying power-range profile, a time-varying power-angleprofile, and/or a time-varying micro-doppler feature) may be provided,as input(s), to the machine learning model.

At block 1410, the electronic device may determine, based at least inpart on the classification of the object, whether to generate an alert.For example, the electronic device may determine that the object is astationary object or a moving object (e.g., and/or whether the object isan visible opaque object or a transparent object such as a window or aglass wall or door) and determine whether to generate the alert bydetermining a velocity of the device relative to the object, and atime-to-impact between the device and the object based on the velocity.In one or more implementations, if the velocity and/or thetime-to-impact satisfy a threshold, the electronic device may determinethat the alert is to be generated, and generate the alert. As discussedherein, an alert generated by the electronic device may include anauditory alert, a tactile alert, and/or a visual alert.

The object information (e.g., classification information, such as anobject type, and/or surface features or other object information)generated by the operations of any of FIG. 12, 13 , or 14 can be used invarious applications. For example, FIG. 15 illustrates a use case inwhich radar data is used for acceleration-gyro drift mitigation in adead-reckoning tracking operation. In the example of FIG. 15 , a user101 is depicted carrying or wearing an electronic device 100 that isequipped with inertial sensors (e.g., inertial sensors 506 of FIG. 5and/or inertial sensor(s) 113 of FIG. 2 ) and one or more wirelesssensing transceivers (e.g. a radar sensor 116, such as a 60 GHz radartransceiver) through a physical environment 1500. In this example, thephysical environment 1500 includes two stationary objects 1502.

The inertial sensor (e.g., one or more gyroscopes, one or moreaccelerometers, and/or one or more magnetometers) may be used to provideestimates of speed and bearing for dead-reckoning tracking of theelectronic device 100. However, in some use cases, due to drift in theseinertial sensors, bias errors can accumulate, thus impacting theaccuracy of estimated track of the electronic device over time. Forexample, FIG. 15 illustrates an estimate track 1504 of the electronicdevice that is increasingly inaccurate due to inertial sensor drift.

In one or more implementations, aspects of the subject technology can beused to mitigate the effect of these inertial sensor drifts by, forexample, detecting, classifying and learning the location and stationarystatus of one or more stationary objects in the environment, and thenusing the stationary objects as reference markers to mitigate the driftin bearing and acceleration.

For example, stationary objects 1502 may be or include fixed obstaclessuch as walls, concrete column beams, cabinets, etc., and can be used asreference points once a location for each object has been determined andonce the objects have been classified as stationary objects. Asdiscussed herein, radar-detected features such as micro-doppler featuresextracted from radar signals 303, can be used to determine that thestationary objects 1502 are stationary, and then range and/or anglemeasurements to the identified markers formed by the stationary objects1502 can be used to reset bearing drifts.

In one or more implementations, a micro-doppler signature from fixedmarkers formed by the stationary objects 1502 can be used to estimate acadence and a ground speed. In one or more implementations, a stridelength of the strides of the user walking through the environment 1500carrying and/or wearing the electronic device 100 can then be estimateddirectly using the radar data, which can also be applied to the inertialdata tracking to help mitigate effect of the inertial sensor drift(e.g., accelerometer drift). In the example of FIG. 15 , a correctedtrack 1506, generated by applying the range and/or angle measurements tothe stationary objects 1502, the cadence and/or ground speed of theuser, and/or the stride length, to the drifting track 1504, is shownthat more closely tracks the actual movements of the user 101.

In the example of FIG. 15 , the radar returns from stationary object1502 may be tracked and classified, surface features, such asmicro-doppler may be extracted, and the speed, cadence, and/or relativeangle of the electronic device 100 to the stationary objects 1502 may beused to mitigate biases in inertial sensors.

In one or more implementations, object information and/or userinformation derived from the radar data obtained using the radar sensor116 can also be applied in other use cases. These other use casesinclude, as non-limiting examples, using the user's directly measuredstride length to measure a distance traveled while walking and/orrunning, a number of calories burned during walk or a run, a number ofsteps taken during a period of time, or other measurements and/orestimates of health-related data for the user.

In accordance with one or more implementations, the subject technologyprovides for use of wireless signal features obtained from a wirelesstransceiver to classify targets/objects. In accordance with one or moreimplementations, time-varying signatures in radar data may be used toclassify target objects. For example, as a transceiver approaches anobject, multiple reflected signals may be captured, and time variationsof the captured reflected signals due to multipath fluctuations canreveal the underlying physical structure of the object. For example, atime-varying radar cross section (RCS) may be used as a feature for theclassification of the detected target/object. In this example, as thetransceiver approaches a target, the estimated RCS may vary in anobject-specific manner due to constructive/destructive combinations ofthe multipath reflections from the target.

In accordance with one or more implementations, an electronic devicehaving a radar sensor may also include an inertial sensor to improve thedetection and classification of targets of interest. For example, in animplementation in which an electronic device has a radar sensor and aninertial sensor, the inertial sensor may be used to identify whether theuser's head is facing a wall or looking downward, which may be used toidentify objects that reflect the radar signals from the electronicdevice. Integration of this information with the extracted wireless(radar) features can provide a more accurate object detection andclassification.

In one or more implementations, an electronic device may be providedwith a machine learning module for classification of targets, which canimprove the user experience for a given application. In one or moreimplementations, an electronic device may provide the relative speedbetween one or more objects and the user/transceiver. The relative speedcan be used to provide a time-to-impact alert. The ability to use radarsensors to provide a time-to-impact alert can provide an improvementover IR-based cameras, particularly in the cases in which theapproaching object is or includes highly transparent and/or reflectivesurfaces. Further, unlike IR-based techniques, an electronic devicehaving a radar sensor for object detection and/or classification canprovide object detection, classification, and/or other features such astime-to-impact alerts in the absence of ambient light.

In one or more implementations, radar reflections from a referencesurface can also be used to determine whether approaching obstacles aremoving or fixed. For example, in an example in which a person is walkingwith a smartphone that includes a radar sensor, the micro-doppler ofradar reflections coming from the ground directly beneath the smartphonecan be provide a “self” micro-doppler view which can be compared withthe micro-doppler of incoming targets. Higher correlation ofmicro-doppler signature of objects with the micro-doppler of the groundmay indicate a fixed target.

In one or more implementations, integration of indoor object detectionand classification using radar signals can provide estimates of stridelength. For example, the extracted micro-doppler signature fromdetected/classified reference objects may be used to estimate a cadencein addition to a velocity.

In one or more implementations, the subject technology provides systems,devices, methods and techniques to process wireless signal reflectionsfrom objects in an environment to extract features that classify thedetected objects. Accurate classification using these wireless signalreflections enables a myriad of wireless sensing applications. In oneexample, a vision impaired person can use a smartphone equipped withwireless sensing system (e.g. a mmWave radar) to navigate an indoorenvironment. In one or more implementations, the subject technology canassist a vision impaired person to navigate indoor environments byproviding an indication and warning of proximity and type of objects(e.g. wall, human, pole) in the path of the user. A warning or alert tothe user of a proximal object or an imminent impact can be providedusing haptic and/or auditory feedback in one or more implementations.Additionally or alternatively, an alert can be graphically presented,e.g., as an image or notification. In still other implementations, analert can be signaled by pausing or stopping audio and/or visual output.

As discussed herein, some implementations of IR depth sensing technologybased on IR-sensors can have problems in detecting transparent, highlyreflective and/or uniform surfaces such as glass doors, windows, mirrorsand uniformly colored walls. In one or more implementations, the subjecttechnology can augment IR depth sensors in detecting these surfaces,estimating the range to the surfaces from the sensor, and classifyingthe type of the surface.

In one or more implementations, the subject technology can be used ingenerating maps of a physical environment of an electronic device. Mapsgenerated using radar sensors to detect, locate, classify, and/or trackobjects in the environment can be provided for use in augmented realityand/or virtual reality applications. The mapping of the environmentusing an electronic device having a radar sensor can provide an accuraterange to objects. In addition, the type of object can be classified toassist the accuracy/efficacy of the mapping.

Various processes defined herein consider the option of obtaining andutilizing a user's personal information. For example, such personalinformation may be utilized in order to provide object tracking and/orclassification. However, to the extent such personal information iscollected, such information should be obtained with the user's informedconsent. As described herein, the user should have knowledge of andcontrol over the use of their personal information.

Personal information will be utilized by appropriate parties only forlegitimate and reasonable purposes. Those parties utilizing suchinformation will adhere to privacy policies and practices that are atleast in accordance with appropriate laws and regulations. In addition,such policies are to be well-established, user-accessible, andrecognized as in compliance with or above governmental/industrystandards. Moreover, these parties will not distribute, sell, orotherwise share such information outside of any reasonable andlegitimate purposes.

Users may, however, limit the degree to which such parties may access orotherwise obtain personal information. For instance, settings or otherpreferences may be adjusted such that users can decide whether theirpersonal information can be accessed by various entities. Furthermore,while some features defined herein are described in the context of usingpersonal information, various aspects of these features can beimplemented without the need to use such information. As an example, ifuser preferences, account names, and/or location history are gathered,this information can be obscured or otherwise generalized such that theinformation does not identify the respective user.

In accordance with aspects of the subject disclosure, a method isprovided that includes obtaining radar signals from a radar sensor of anelectronic device; identifying a motion characteristic corresponding tothe electronic device based on the radar signals; detecting an object inan environment of the electronic device using the radar signals; andclassifying the object using the radar signals and the identified motioncharacteristic; and determining, by the electronic device, whether togenerate an alert based on the detecting and classifying of the object.

In accordance with aspects of the subject disclosure, a method isprovided that includes obtaining radar signals from a radar sensor of anelectronic device; extracting a radar cross-section and a micro-dopplersignal from the radar signals; classifying an object in an environmentof the electronic device based on the radar cross-section and themicro-doppler signal; and determining, by the electronic device andbased at least in part on the classification of the object, whether togenerate an alert.

In accordance with aspects of the subject disclosure, a portableelectronic device is provided that includes a radar sensor and one ormore processors configured to: obtain a radar signal from the radarsensor; identify a target of interest in an environment of the portableelectronic device using the radar signal; extract a time-varying surfacefeature, for the target of interest, from the radar signals; obtain aclassification of an object corresponding to the target of interestusing the extracted time-varying surface feature; and determine whetherto generate an alert based at least in part on the classification of theobject.

Implementations within the scope of the present disclosure can bepartially or entirely realized using a tangible computer-readablestorage medium (or multiple tangible computer-readable storage media ofone or more types) encoding one or more instructions. The tangiblecomputer-readable storage medium also can be non-transitory in nature.

The computer-readable storage medium can be any storage medium that canbe read, written, or otherwise accessed by a general purpose or specialpurpose computing device, including any processing electronics and/orprocessing circuitry capable of executing instructions. For example,without limitation, the computer-readable medium can include anyvolatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM,and TTRAM. The computer-readable medium also can include anynon-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM,NVRAM, flash, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM,NRAM, racetrack memory, FJG, and Millipede memory.

Further, the computer-readable storage medium can include anynon-semiconductor memory, such as optical disk storage, magnetic diskstorage, magnetic tape, other magnetic storage devices, or any othermedium capable of storing one or more instructions. In one or moreimplementations, the tangible computer-readable storage medium can bedirectly coupled to a computing device, while in other implementations,the tangible computer-readable storage medium can be indirectly coupledto a computing device, e.g., via one or more wired connections, one ormore wireless connections, or any combination thereof.

Instructions can be directly executable or can be used to developexecutable instructions. For example, instructions can be realized asexecutable or non-executable machine code or as instructions in ahigh-level language that can be compiled to produce executable ornon-executable machine code. Further, instructions also can be realizedas or can include data. Computer-executable instructions also can beorganized in any format, including routines, subroutines, programs, datastructures, objects, modules, applications, applets, functions, etc. Asrecognized by those of skill in the art, details including, but notlimited to, the number, structure, sequence, and organization ofinstructions can vary significantly without varying the underlyinglogic, function, processing, and output.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, one or more implementationsare performed by one or more integrated circuits, such as ASICs orFPGAs. In one or more implementations, such integrated circuits executeinstructions that are stored on the circuit itself.

Those of skill in the art would appreciate that the various illustrativeblocks, modules, elements, components, methods, and algorithms describedherein may be implemented as electronic hardware, computer software, orcombinations of both. To illustrate this interchangeability of hardwareand software, various illustrative blocks, modules, elements,components, methods, and algorithms have been described above generallyin terms of their functionality. Whether such functionality isimplemented as hardware or software depends upon the particularapplication and design constraints imposed on the overall system.Skilled artisans may implement the described functionality in varyingways for each particular application. Various components and blocks maybe arranged differently (e.g., arranged in a different order, orpartitioned in a different way) all without departing from the scope ofthe subject technology.

It is understood that any specific order or hierarchy of blocks in theprocesses disclosed is an illustration of example approaches. Based upondesign preferences, it is understood that the specific order orhierarchy of blocks in the processes may be rearranged, or that allillustrated blocks be performed. Any of the blocks may be performedsimultaneously. In one or more implementations, multitasking andparallel processing may be advantageous. Moreover, the separation ofvarious system components in the implementations described above shouldnot be understood as requiring such separation in all implementations,and it should be understood that the described program components andsystems can generally be integrated together in a single softwareproduct or packaged into multiple software products.

As used in this specification and any claims of this application, theterms “base station”, “receiver”, “computer”, “server”, “processor”, and“memory” all refer to electronic or other technological devices. Theseterms exclude people or groups of people. For the purposes of thespecification, the terms “display” or “displaying” means displaying onan electronic device.

As used herein, the phrase “at least one of” preceding a series ofitems, with the term “and” or “or” to separate any of the items,modifies the list as a whole, rather than each member of the list (i.e.,each item). The phrase “at least one of” does not require selection ofat least one of each item listed; rather, the phrase allows a meaningthat includes at least one of any one of the items, and/or at least oneof any combination of the items, and/or at least one of each of theitems. By way of example, the phrases “at least one of A, B, and C” or“at least one of A, B, or C” each refer to only A, only B, or only C;any combination of A, B, and C; and/or at least one of each of A, B, andC.

The predicate words “configured to”, “operable to”, and “programmed to”do not imply any particular tangible or intangible modification of asubject, but, rather, are intended to be used interchangeably. In one ormore implementations, a processor configured to monitor and control anoperation or a component may also mean the processor being programmed tomonitor and control the operation or the processor being operable tomonitor and control the operation. Likewise, a processor configured toexecute code can be construed as a processor programmed to execute codeor operable to execute code.

Phrases such as an aspect, the aspect, another aspect, some aspects, oneor more aspects, an implementation, the implementation, anotherimplementation, some implementations, one or more implementations, anembodiment, the embodiment, another embodiment, some implementations,one or more implementations, a configuration, the configuration, anotherconfiguration, some configurations, one or more configurations, thesubject technology, the disclosure, the present disclosure, othervariations thereof and alike are for convenience and do not imply that adisclosure relating to such phrase(s) is essential to the subjecttechnology or that such disclosure applies to all configurations of thesubject technology. A disclosure relating to such phrase(s) may apply toall configurations, or one or more configurations. A disclosure relatingto such phrase(s) may provide one or more examples. A phrase such as anaspect or some aspects may refer to one or more aspects and vice versa,and this applies similarly to other foregoing phrases.

The word “exemplary” is used herein to mean “serving as an example,instance, or illustration”. Any embodiment described herein as“exemplary” or as an “example” is not necessarily to be construed aspreferred or advantageous over other implementations. Furthermore, tothe extent that the term “include”, “have”, or the like is used in thedescription or the claims, such term is intended to be inclusive in amanner similar to the term “comprise” as “comprise” is interpreted whenemployed as a transitional word in a claim.

All structural and functional equivalents to the elements of the variousaspects described throughout this disclosure that are known or latercome to be known to those of ordinary skill in the art are expresslyincorporated herein by reference and are intended to be encompassed bythe claims. Moreover, nothing disclosed herein is intended to bededicated to the public regardless of whether such disclosure isexplicitly recited in the claims. No claim element is to be construedunder the provisions of 35 U.S.C. § 112(f) unless the element isexpressly recited using the phrase “means for” or, in the case of amethod claim, the element is recited using the phrase “step for”.

The previous description is provided to enable any person skilled in theart to practice the various aspects described herein. Variousmodifications to these aspects will be readily apparent to those skilledin the art, and the generic principles defined herein may be applied toother aspects. Thus, the claims are not intended to be limited to theaspects shown herein, but are to be accorded the full scope consistentwith the language claims, wherein reference to an element in thesingular is not intended to mean “one and only one” unless specificallyso stated, but rather “one or more”. Unless specifically statedotherwise, the term “some” refers to one or more. Pronouns in themasculine (e.g., his) include the feminine and neutral gender (e.g., herand its) and vice versa. Headings and subheadings, if any, are used forconvenience only and do not limit the subject disclosure.

What is claimed is:
 1. A method, comprising: obtaining radar signalsfrom a radar sensor of an electronic device; identifying a motioncharacteristic corresponding to the electronic device based at least inpart on the radar signals; detecting an object in an environment of theelectronic device based at least in part on the radar signals;classifying the object based at least in part on the radar signals andthe identified motion characteristic; and determining, by the electronicdevice, whether to generate an alert based on the detecting andclassifying of the object.
 2. The method of claim 1, wherein the motioncharacteristic comprises a characteristic of walking motion.
 3. Themethod of claim 2, wherein the motion characteristic further comprises acharacteristic of at least one of a leg swing motion or an arm swingmotion.
 4. The method of claim 2, wherein classifying the objectcomprises classifying the object as a moving object or a stationaryobject, and wherein identifying the motion characteristic comprisesidentifying, using the radar signals, a first cadence corresponding tothe walking motion.
 5. The method of claim 4, wherein the objectcomprises the stationary object, and wherein classifying the objectcomprises: identifying, using the radar signals, a second cadencecorresponding to the object; and determining that the second cadencesubstantially matches the first cadence.
 6. The method of claim 5,wherein identifying the first cadence using the radar signals comprisesidentifying the first cadence using a first portion of the radar signalscorresponding to a reflection from a ground surface, and whereinidentifying the second cadence using the radar signals comprisesidentifying the second cadence using a second portion of the radarsignals corresponding to a reflection from the object, the object beingdifferent from the ground surface.
 7. The method of claim 5, furthercomprising: tracking motion of the electronic device using a sensor ofthe electronic device other than the radar sensor; determining alocation of the stationary object using the radar signals; and modifyingthe tracking of the motion of the electronic device based on thelocation of the stationary object.
 8. The method of claim 5, wherein thestationary object comprises a pane of glass, wherein classifying theobject further comprises classifying the object as glass, and, whereinidentifying the motion characteristic further comprises determining avelocity of the electronic device relative to the pane of glass.
 9. Themethod of claim 8, wherein determining whether to generate the alertcomprises estimating a time-to-impact between the electronic device andthe pane of glass based on the velocity.
 10. The method of claim 9,wherein determining whether to generate the alert comprises determiningthat the time-to-impact satisfies a threshold for generating the alert,the method further comprising generating the alert with the electronicdevice.
 11. The method of claim 4, wherein the object comprises themoving object, and wherein classifying the object comprises:identifying, using the radar signals, a second cadence corresponding tothe object; and determining that the second cadence is different fromthe first cadence, wherein identifying the first cadence using the radarsignals comprises identifying the first cadence using a first portion ofthe radar signals corresponding to a reflection from a ground surface,and wherein identifying the second cadence using the radar signalscomprises identifying the second cadence using a second portion of theradar signals corresponding to a reflection from the object.
 12. Themethod of claim 4, further comprising: determining a stride length of adevice user based in part on the first cadence; and generating healthdata for the device user based on the stride length.
 13. A method,comprising: obtaining radar signals from a radar sensor of an electronicdevice; extracting a radar cross-section and a micro-doppler signal fromthe radar signals; classifying an object in an environment of theelectronic device based on the radar cross-section and the micro-dopplersignal; and determining, by the electronic device and based at least inpart on the classification of the object, whether to generate an alert.14. The method of claim 13, wherein classifying the object based on theradar cross-section and the micro-doppler signal comprises classifyingthe object based on a time variation of the radar cross-section and atime variation of the micro-doppler signal.
 15. The method of claim 14,wherein classifying the object based on the time variation of the radarcross-section and the time variation of the micro-doppler signalcomprises distinguishing between an opaque planar surface and atransparent planar surface at least in part by providing at least thetime variation of the radar cross-section to a machine learning engineat the electronic device.
 16. The method of claim 13, furthercomprising, prior to classifying the object: performing an initialtarget detection using the radar signals; determining a location of apotential target object based on the initial target detection; andextracting the radar cross-section, the micro-doppler signal, a range,and an angle, from the radar signals based on the location, in part, byperforming a beam forming operation on the radar signals using thelocation.
 17. A portable electronic device, comprising: a radar sensor;and one or more processors configured to: obtain a radar signal from theradar sensor; identify a target of interest in an environment of theportable electronic device using the radar signal; extract atime-varying surface feature, for the target of interest, from the radarsignals; obtain a classification of an object corresponding to thetarget of interest using the extracted time-varying surface feature; anddetermine whether to generate an alert based at least in part on theclassification of the object.
 18. The portable electronic device ofclaim 17, further comprising an inertial sensor, wherein the one or moreprocessors are configured to identify the target of interest using theradar signal and sensor data from the inertial sensor of the portableelectronic device.
 19. The portable electronic device of claim 18,wherein the time-varying surface feature comprises at least one of atime-varying radar cross-section, a time-varying micro-doppler feature,a time-varying power-range profile, or a time-varying power-angleprofile, wherein the time-varying power-angle profile may include atime-varying power with respect to at least one of an azimuth or anelevation.
 20. The portable electronic device of claim 17, wherein theportable electronic device comprises a handheld electronic device or awearable electronic device further comprising a memory storing a machinelearning model trained to classify objects based on time-varying radarcross-sections, wherein the extracted time-varying surface featurecomprises a time-varying radar cross-section, and wherein the one ormore processors are configured to obtain the classification of theobject corresponding to the target of interest using the extractedtime-varying surface feature by providing a time variation of a radarcross-section to the machine learning model, and obtaining a resultingoutput from the machine learning model.